Tag: LLM inference energy
3 reviews
Smoothing the power curve of LLM training and inference
Which kernel- and scheduler-level techniques reduce power ripples during LLM training and inference, and at what cost in energy and throughput?
GPU power draw during LLM training and inference is bursty: compute-bound prefill spikes, memory-bound decode idles far below the power ceiling, and synchronised training steps add further lumps. Across 89 sources (2011-2026) the evidence shows GPU frequency and power-capping settings have pronounced energy sweet spots (routinely 15-35% energy saved for single-digit throughput loss) that apply only to compute-bound phases; the newest work shows the power ramp between scheduling steps, not the peak, is the controllable quantity. Confidence is moderate: device-level trade-offs are replicated across many groups, while the ripple-smoothing literature is young (mostly 2025-2026 preprints) and largely single-node or emulation-scale.
Benchmarking tokens per watt: how AI inference energy efficiency is measured
What benchmarks and measurement methodologies exist for quantifying AI inference energy efficiency in tokens per watt (or per joule), from accelerator to data-centre level, and what do they establish?
The literature on measuring AI inference efficiency in tokens-per-watt terms is young (mostly 2023-2026) and fragmented: one consortium standard exists at the system level (MLPerf Power), but the benchmarks that actually report tokens per watt or joules per token are research tools with incompatible measurement boundaries. Measured numbers span orders of magnitude — roughly 3-4 joules per output token for a 65B model on A100s, 0.002 to 2.9 kWh per 1,000 inferences depending on task, a 65x spread across models in commercial data centres, and a proposed 1/W law under which tokens per watt halves each time the context window doubles. No retrieved benchmark measures tokens per watt at the data-centre (facility) level; facility efficiency is still expressed as PUE, so wall-level tokens per watt is derived, not measured. Confidence is moderate: 20 of 65 sources were read in full text and several prominent items were unreachable in-session.
Energy efficiency across the AI datacenter stack
What does the 2023-2026 literature establish about improving energy efficiency across the AI datacenter stack - from GPU power management and workload scheduling to grid-level coordination - and how strong is the evidence for each lever?
AI datacenters are among the fastest-growing electricity consumers of the 2020s, and the 2023-2026 literature offers a stack of levers to improve their efficiency: GPU power capping and frequency control, energy- and carbon-aware scheduling, LLM serving configuration, cooling control, and grid-interactive operation. Evidence is strongest at the device and workload levels (measured savings of roughly 15-50% on specific workloads), moderate for scheduling (mostly simulation), and thinnest at grid level, where a single production deployment carries much of the weight. The key caveat: nearly every lever is workload- and hardware-generation-specific - power capping that saves ~23% energy on one model hurts SLOs on another - so none generalises without measurement.